Skip to content

chore: bump kubedog to fail fast on ReplicaSet pod-create errors - #780

Merged
ilya-lesikov merged 3 commits into
mainfrom
chore/bump-kubedog-replicaset-failures
Oct 29, 2026
Merged

ilya-lesikov merged 3 commits into
mainfrom
chore/bump-kubedog-replicaset-failures

Conversation

@ilya-lesikov

@ilya-lesikov ilya-lesikov commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Summary

Bumps github.com/werf/kubedog to the merge commit of werf/kubedog#404 on kubedog main (97f4f13), which makes Deployment tracking fail fast on ReplicaSet pod-create errors such as exceeded quota (werf/kubedog#398).

werf/kubedog#404 is merged; this PR pins its merge commit.

What

  • go.mod/go.sum: github.com/werf/kubedog v0.13.1-0.20260915153156-438ef109f804 → v0.13.1-0.20261029011008-97f4f131413e.
  • The bump also brings in the kubedog main commits since the previous pin (438ef10..b201189): golangci-lint cleanups and reformatting. They do not change behaviour.
  • It also brings in hardening from fix: fail Deployment tracking fast on ReplicaSet pod-create errors kubedog#404: quota quantities parsed from a FailedCreate message are bounded, so a crafted ReplicaSet status cannot keep the tracker busy.

Why

Without this bump, nelm (and werf) keep waiting for the tracking timeout when a Deployment's new ReplicaSet cannot create pods. The failure is only visible in ReplicaSet events.

Behaviour changes for nelm users

These come from pkg/dyntracker, which pkg/plan/plan_execute.go uses:

  • An invalid pod spec, or a quota deadlock the rollout cannot resolve on its own, now fails the resource immediately, regardless of werf.io/failures-allowed-per-replica. A quota deadlock is reported about 10s after it is observed, with the FailedCreate ... exceeded quota diagnostic.
  • Other pod-create failures, including quota failures that the rollout can recover from, are added once as errors against the failures budget. Before, they were not seen at all.
  • With werf.io/fail-mode: IgnoreAndContinueDeployProcess, these fatal failures do not fail the deploy.
  • Only ReplicaSets controlled by the Deployment are treated as its new ReplicaSet. An orphan ReplicaSet that matches the template is ignored until the Deployment adopts it.
  • A Deployment failure from an event is charged to the budget once, not again on every later status update.
  • Fast-failing on a quota deadlock needs list pods (already needed for tracking) and get resourcequotas in the namespace. The built-in view, edit and admin roles have both. Without them, quota failures are counted rather than fatal.

Testing

  • task build, task test:unit (67 suites) and task lint pass.
  • The kubedog PR has unit tests for each case and was verified on kind with a ResourceQuota, tracking through dyntracker as nelm does.

@ilya-lesikov
ilya-lesikov merged commit 109bfb4 into main Oct 29, 2026
8 checks passed
@ilya-lesikov
ilya-lesikov deleted the chore/bump-kubedog-replicaset-failures branch October 29, 2026 01:45
This was referenced Oct 30, 2026
dmmordvi added a commit that referenced this pull request Nov 3, 2026
## Summary

Bumps `github.com/werf/kubedog` to the merge commit of werf/kubedog#406
on kubedog `main` (`1ed920d`). A long-lived process that runs `nelm`
release actions over and over with tracking (for example, a controller
that deploys a release on every reconcile) no longer accumulates
informers that hold the cache of a resource type for a whole namespace
after each action ends. Before, the memory such a process held grew with
every deploy until it restarted.

## What

- `go.mod`/`go.sum`: `github.com/werf/kubedog`
`v0.13.1-0.20261029011008-97f4f131413e` →
`v0.13.1-0.20261103160714-1ed920dee18b`. The range `97f4f13..1ed920d` is
the single commit of werf/kubedog#406.
- After a release install, upgrade, uninstall or rollback with tracking
finishes and its context is canceled, the shared informers it started
stop and release their caches, even when the trackers stopped reading
before an informer delivered all its events. Applies to the Pod,
Deployment, StatefulSet, DaemonSet, Job, Canary and ReplicaSet trackers,
the generic resource state and events watchers and the event informer
that `pkg/dyntracker` uses.
- A fatal watch error raised after the action finished no longer blocks
the informer: it is dropped once the informer factory is stopped instead
of waiting forever on the watch error channel.
- When a Job, DaemonSet, StatefulSet or Pod tracker fails halfway
through starting, the informers it already started are released.
- While tracking runs, every event is delivered as before: channel
capacities and send order are unchanged, nothing is dropped. Tracking
results, progress output and errors do not change.
- Library API: kubedog adds the exported helper `utils.SendOrDone(ctx,
ch, v)`. Nothing in the `nelm` API changes.
- `go.mod`: `golang.org/x/sync v0.20.0` moves from the indirect to the
direct block, same version. `pkg/resource/spec/patches_debug_test.go`
(#785) already imports it directly; `go mod tidy` on `main` without this
bump makes the same change.
- UNVERIFIED: not reproduced against a real cluster. kubedog covers
every changed handler with a regression test on a fake dynamic client
that shows the informer goroutines stay alive without the fix and exit
with it.

## Why

Informer event handlers run in the client-go listener goroutine and sent
to tracker channels without watching for cancellation. When a handler
had more events to deliver than the channel could buffer at the moment
the tracker stopped reading (e.g. the initial Add events for several
Pods of a Deployment that is already ready), the send blocked forever,
the informer stayed in `wg.Wait()` after the stop signal, and its
indexer stayed referenced. 38643e9 in kubedog (brought in by #780) fixed
this for the ReplicaSet informer only. Bigger channel buffers only move
the threshold, since the number of objects an informer replays on start
is unbounded.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant